Bump minimal supported Python version to `py3.11` by xsedla1o · Pull Request #40 · CESNET/dp3

xsedla1o · 2026-05-13T12:01:05Z

Python 3.9 has reached EOL and Python 3.10 EOL is approaching, so this branch bumps DP3’s supported/runtime baseline to Python 3.11 and modernizes the codebase/tooling accordingly.

Key changes

Require Python 3.11 in packaging, Ruff/Black targets, Docker images, CI docs deploy, README, and docs.
Apply Ruff/Black modernization for Python 3.11, including PEP 604 union types and related typing/import cleanup.
Fix Python 3.11/runtime typing issues around dynamic API entity ID validation and MACAddress.
Replace deprecated Pydantic APIs with current equivalents.
Preserve /entity/{etype}/{eid}/set/{attr} behavior by explicitly parsing JSON request bodies.
Add small typing cleanups for config defaults and task queue pyright compatibility.

Copilot

Pull request overview

This PR raises DP3’s Python baseline to 3.11 and modernizes typing/tooling accordingly, including updates to runtime/API code paths that rely on Pydantic validation and datetime handling.

Changes:

Bump declared/runtime Python version to 3.11 across packaging, tooling targets, Docker images, CI deploy, and docs.
Modernize typing (PEP 604 unions, collections.abc imports), and update Pydantic v2 usage (model_validate, model_dump).
Adjust API/runtime behavior around entity ID validation and JSON request body parsing for /entity/{etype}/{eid}/set/{attr}.

Reviewed changes

Copilot reviewed 58 out of 63 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
tests/test_example/dps_gen.py	Switch to `datetime.UTC` for UTC timestamps in test generator.
tests/test_example/dps_gen_realtime.py	Switch to `datetime.UTC` and remove local UTC alias.
tests/test_common/test_types.py	Update UTC expectations to use `datetime.UTC`.
tests/test_common/test_snapshots.py	Typing modernizations (PEP 604, `collections.abc.Callable`), UTC import cleanup.
tests/test_common/test_module_testing.py	Use `datetime.UTC` instead of project UTC alias.
tests/test_common/test_magic.py	Use `datetime.UTC` in expected datetime values.
tests/test_api/test_telemetry.py	Use `datetime.UTC` in API tests.
tests/test_api/test_snapshots.py	Use `datetime.UTC` in API tests.
tests/test_api/test_raw.py	Use `datetime.UTC` in API tests.
tests/test_api/test_get_entity_eid_data.py	Use `datetime.UTC` in API tests.
tests/test_api/test_01_datapoints.py	Use `datetime.UTC` in API tests.
tests/test_api/common.py	Prefer `collections.abc.Callable` over `typing.Callable`.
requirements.txt	Relax some patch pins while keeping compatible minor versions.
requirements.scripts.txt	Update pandas major line (2.2).
README.md	Document Python 3.11+ prerequisites.
pyproject.toml	Set `requires-python >=3.11` and update Black/Ruff targets to py311.
dp3/testing/registrar.py	Typing modernizations in test registrar APIs.
dp3/testing/config.py	Typing modernizations (PEP 604 optionals).
dp3/testing/case.py	Typing modernizations; use `datetime.UTC`.
dp3/template/app/docker/python/Dockerfile	Bump template Python image to 3.11-slim.
dp3/task_processing/task_queue.py	Typing updates, thread liveness fixes, small robustness tweaks.
dp3/task_processing/task_hooks.py	Prefer `collections.abc.Callable`.
dp3/task_processing/task_executor.py	Prefer `collections.abc.Callable`.
dp3/snapshots/snapshot_hooks.py	Typing modernizations for hook signatures.
dp3/snapshots/snapshooter.py	Typing modernizations; migrate to `model_dump`; zip strictness.
dp3/scripts/dummy_sender.py	Switch to `datetime.UTC` and remove local UTC alias.
dp3/scripts/datapoint_log_converter.py	Typing import cleanup (`collections.abc.Callable`).
dp3/scripts/add_hashes.py	Replace deprecated Pydantic `parse_obj` with `model_validate`.
dp3/history_management/telemetry.py	Use `datetime.UTC`; typing modernizations.
dp3/history_management/history_manager.py	Migrate Pydantic extra handling to `ConfigDict`; typing modernizations.
dp3/database/snapshots.py	Typing modernizations (PEP 604 unions).
dp3/database/schema_cleaner.py	Use `datetime.UTC`; typing import cleanup.
dp3/database/magic.py	Use `datetime.UTC`; typing modernizations for helper return types.
dp3/database/database.py	Use `datetime.UTC`; typing modernizations; zip strictness.
dp3/database/config.py	Typing modernizations for discriminated union config.
dp3/core/updater.py	Use `datetime.UTC`; typing import cleanup.
dp3/core/link_manager.py	Use `datetime.UTC`.
dp3/core/collector.py	Use `datetime.UTC`.
dp3/common/utils.py	Typing modernizations for duration parsing signature.
dp3/common/types.py	Use `datetime.UTC`; PEP 604 typing cleanup.
dp3/common/task.py	Typing modernizations for validator signatures and fields.
dp3/common/scheduler.py	Typing modernizations for scheduler API.
dp3/common/mac_address.py	Add `__future__.annotations`; modernize Pydantic core schema typing.
dp3/common/entityspec.py	Typing modernizations for discriminated union fields.
dp3/common/datatype.py	Typing modernizations for primitive types and aliases.
dp3/common/datapoint.py	Typing modernizations; PEP 604 unions for aliases/fields.
dp3/common/control.py	Prefer `collections.abc.Callable`.
dp3/common/config.py	Pydantic `ConfigDict` migration; improve typing for `HierarchicalDict.get`.
dp3/common/callback_registrar.py	Typing modernizations for hook signatures and scheduler registration.
dp3/common/attrspec.py	Typing modernizations (PEP 604 unions) for config models and aliases.
dp3/bin/shcmd/entity/common.py	Typing modernizations for optional return types/params.
dp3/bin/shcmd/entity/init.py	Typing modernizations for return types.
dp3/bin/shcmd/common.py	Typing modernizations for API client helpers and completion loading.
dp3/bin/check.py	Use `zip(..., strict=False)` for safer zips.
dp3/api/routers/telemetry.py	Typing modernizations for optional query params.
dp3/api/routers/entity.py	Update entity ID parsing typing; explicitly parse JSON body for set-attr endpoint; use `model_dump`.
dp3/api/internal/models.py	Rework dynamic EntityId model generation and TypeAdapter union typing.
dp3/api/internal/entity_response_models.py	Typing modernizations for response model aliases.
dp3/api/internal/config.py	Replace deprecated Pydantic `parse_obj` with `model_validate`.
docs/howto/get-started.md	Document Python 3.11+ prerequisites.
docs/howto/develop-dp3.md	Document Python 3.11+ prerequisites.
docker/python/Dockerfile	Bump runtime Python image to 3.11-slim.
.github/workflows/deploy.yml	Use Python 3.11 for docs deploy workflow.

Comments suppressed due to low confidence (1)

dp3/snapshots/snapshooter.py:582

master_record[datapoint.attr].append() is called without an argument, which will raise TypeError and break snapshot generation. This likely should append the newly created dp_dict (and should preserve the same structure used when initializing the list in the else branch).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

dbnk0

LGTM

xsedla1o changed the title ~~Dp3 py311~~ Bump minimal supported Python version to py3.11 May 13, 2026

xsedla1o added 9 commits May 21, 2026 16:05

API: parse set attribute JSON body explicitly

1b2a495

Python: require 3.11 and modernize typing

76dcd61

Docs: update Python prerequisite to 3.11

9faa593

API: separate EntityId validation type

bb89590

Pydantic: replace deprecated APIs

4a27e01

Config: type hierarchical get default

9f41f65

Typing: satisfy pyright for task queue

1b5234b

Typing: import UTC from datetime

29b032b

requirements: relax specifiers

c603639

xsedla1o force-pushed the dp3-py311 branch from 291a4eb to c603639 Compare May 21, 2026 14:14

xsedla1o assigned martinzadnik and dbnk0 and unassigned martinzadnik May 21, 2026

xsedla1o requested a review from Copilot May 21, 2026 14:15

Copilot started reviewing on behalf of xsedla1o May 21, 2026 14:16 View session

xsedla1o marked this pull request as ready for review May 21, 2026 14:16

Copilot AI reviewed May 21, 2026

View reviewed changes

xsedla1o added 2 commits May 21, 2026 16:48

API: fix no entity configured edge-case

95dfd41

Logging: suppress dependency loggers

1c9797c

xsedla1o force-pushed the dp3-py311 branch from dd8e1ba to 1c9797c Compare May 21, 2026 15:05

dbnk0 approved these changes May 21, 2026

View reviewed changes

xsedla1o merged commit 22888b3 into master May 21, 2026
4 checks passed

xsedla1o deleted the dp3-py311 branch May 21, 2026 15:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bump minimal supported Python version to `py3.11`#40

Bump minimal supported Python version to `py3.11`#40
xsedla1o merged 11 commits into
masterfrom
dp3-py311

xsedla1o commented May 13, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dbnk0 left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

xsedla1o commented May 13, 2026

Key changes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dbnk0 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants